Crate ress

ress A crate for parsing raw JS into a token stream

The primary interfaces are the function tokenize and the struct Scanner. The Scanner struct impls Iterator and the tokenize function is just a wrapper around Scanner::collect().

The Scanner will provide a stream of Items, and Item is has 3 properties a Token, a Span, and a SourceLocation. The Span is a representation of where the Item exists in the original source while the Token provides details about what JavaScript token it represents.

Modules

• error
• prelude
• tokens

Structs

• Item
  A single token with additional metadata
• JSBuffer
• ManualScanner
• ManualState
  All of the important state for the scanner, used to cache and reset a Scanner
• Position
  A single character position in the file including the line/column number
• Scanner
  The primary interface of this crate used to tokenize any JS text into a stream of Items.
• ScannerState
  All of the important state for the scanner, used to cache and reset a Scanner
• SourceLocation
  The start and end position of a token including the line/column number
• Span
  The start and end of a token as the byte index in the original text
• Tokenizer
  This structure will perform the low level tokenization before the Scanner provides additional context

Enums

• OpenCurlyKind
  For keeping track of the nested-ness of templates and blocks

Functions

• tokenize
  a convince function for collecting a scanner into a Vec<Token>